Production apparatus for index information with link information, production apparatus for image data with tag information, production method for index information with link information, production method for image data with tag information and recording medium

ABSTRACT

Disclosed herein is a production apparatus for index information with link information. The apparatus includes a database in which a corresponding relationship between region identification information applied to a key region which provides a partial region in a main image specified arbitrarily by an operator and link information for accessing to related information of the partial region is listed. The apparatus further includes an information embedding section configured to refer to said database to embed, into index information in which a corresponding relationship between the region identification information applied to the key region and time information is listed, the corresponding link information.

CROSS REFERENCES TO RELATED APPLICATIONS

The present invention contains subject matter related to Japanese PatentApplication JP 2005-198172, filed in the Japanese Patent Office on Jul.6, 2005, the entire contents of which being incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to an apparatus and method for producing indexinformation with link information wherein link information is embeddedin a key region specified arbitrarily by an operator. The presentinvention further relates to an apparatus and method for producing imagedata wherein, in response to a click operation of a partial region inwhich tag information is embedded, information associated with thepartial region can be displayed on a screen. The present inventionrelates also to a recording medium in which a program for implementingthe functions mentioned is recorded.

2. Description of Related Art

Together with setup of the communication infrastructures and increase ofthe recording capacity in recent years, it is demanded to place variousservices proposed heretofore into practical use. The technique describedis disclosed, for example, in Japanese Patent Laid-open No. Hei7-231308.

SUMMARY OF THE INVENTION

However, a technique has not been established which can apply taginformation to a particular partial region in moving pictures through arealistic amount of work.

The inventors of the present invention propose a technique which canapply tag information to a main image through a realistic amount ofwork.

(A) Index Information with Link Information

First, the inventors of the present invention propose, as an apparatuswhich produces index information with link information, apparatus whichhave the following processing functions:

(a) Mode 1

(a1) a database in which a corresponding relationship between regionidentification information applied to a key region which provides apartial region in a main image specified arbitrarily by an operator andlink information for accessing to related information of the partialregion is listed; and

(a2) a processing function configured to refer to the database to embed,into index information in which a corresponding relationship between theregion identification information applied to the key region and timeinformation is listed, the corresponding link information.

(b) Mode 2

(b1) a database in which a corresponding relationship between objectidentification information applied to a key region which provides apartial region in a main image specified arbitrarily by an operator andlink information for accessing to related information of the partialregion is listed; and

(b2) a processing function configured to refer to the database to embed,into index information in which a corresponding relationship between theobject identification information applied to the key region and timeinformation is listed, the corresponding link information.

(B) Image Data with Tag Information

Further, the inventors of the present invention propose, as a productionapparatus for image data which can display, in response to a clickoperation of a particular region in which tag information is embedded,information associated with the particular portion on a screen,apparatus which have the following processing functions:

(a) Mode 1

(a1) a database in which a corresponding relationship between regionidentification information applied to a key region which provides apartial region in a main image specified arbitrarily by an operator andlink information for accessing to related information of the partialregion is listed;

(a2) a processing function configured to refer to the database to embed,into index information in which a corresponding relationship between theregion identification information applied to the key region and timeinformation is listed, the corresponding link information; and

(a3) a processing function configured to output the index informationwith link information in which the link information is embedded, themain image and a key image to which the region identificationinformation which specifies the key region in combination with the timeinformation is applied in a mutually associated relationship.

(b) Mode 2

(b1) a database in which a corresponding relationship between objectidentification information applied to a key region which provides apartial region in a main image specified arbitrarily by an operator andlink information for accessing to related information of the partialregion is listed;

(b2) a processing function configured to refer to the database to embed,into index information in which a corresponding relationship between theobject identification information applied to the key region and timeinformation is listed, the corresponding link information; and

(b3) a processing function configured to output the index informationwith link information in which the link information is embedded, themain image and a key image to which the region identificationinformation which specifies the key region in combination with the timeinformation is applied in a mutually associated relationship.

The inventors of the present invention proposes a technique of indexinformation with link information by embedding link information intoindex information in which a corresponding relationship between regionidentification information applied to a key region which forms a keyimage and time information is listed or index information in which acorresponding relationship between object identification informationapplied to a key region which forms a key image and time information islisted.

As far as the corresponding relationship between the regionidentification information and the link information or the correspondingrelationship between the link information and the object identificationinformation is assured, the coordination between the key region and thelink information can be implemented efficiently.

Further, the inventors of the present invention propose a technique ofoutputting index information with link information, a main image and akey image in a mutually associated relationship. Consequently, the usercan cause information associated with a partial region in which taginformation is embedded to be displayed only if the viewer clicks thepartial region on a screen of a main image.

The above and other objects, features and advantages of the presentinvention will become apparent from the following description and theappended claims, taken in conjunction with the accompanying drawings inwhich like parts or elements denoted by like reference symbols.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of a configuration of alinear editing system;

FIGS. 2A to 2C are schematic views showing a corresponding relationshipbetween display examples of a working screen and key images produced;

FIG. 3 is a block diagram showing an example of a functionalconfiguration of a tag information production apparatus;

FIG. 4 is a diagrammatic view illustrating a corresponding relationshipbetween a key image and key information which prescribes taginformation;

FIG. 5 is a similar view but illustrating a corresponding relationshipbetween index information and key information which defines taginformation;

FIG. 6 is a view illustrating an example of index information;

FIG. 7 is a flow diagram illustrating a corresponding relationshipbetween a tagging operation and internal processes executed in responseto the tagging operation;

FIG. 8 is a flow chart illustrating an outline of the substance ofprocessing of a program to be installed into a key information inputtingapparatus;

FIG. 9 is a block diagram showing an example of a configuration of anonlinear editing system;

FIGS. 10A and 10B are schematic views showing processing images whereshoes of an image of a subject are set as key regions;

FIG. 11 is a block diagram showing an example of an internalconfiguration of a key region setting section which implements anautomatic extraction function of a key region based on a colorcondition;

FIG. 12 is a schematic view showing a processing image where a keyregion is set so as to follow up a characteristic amount of each of theshoes of the image of the subject;

FIG. 13 is a block diagram showing an example of an internalconfiguration of a key region setting section where it implements anautomatic following up function of a key region;

FIGS. 14A and 14B are schematic views illustrating a positionalrelationship between a key region set on a working screen selected at astart point and a key region set on another working screen selected atan end point;

FIG. 15 is a schematic view showing a locus where the shoes of thesubject image move linearly between the two different points of time;

FIG. 16 is a block diagram showing another example of an internalconfiguration of the key region setting section where it implements anautomatic following up function of a key region;

FIGS. 17A, 17B, 18A and 18B are schematic views showing examples of theshape of a key region prepared in fixed shapes;

FIGS. 19A and 19B are schematic views illustrating a relationshipbetween the size of a main image and the size of a key image;

FIGS. 20A and 20B are diagrammatic views showing embedded images of linkinformation in index information;

FIG. 21 is a diagrammatic view showing an example of an internalconfiguration of a link information embedding section;

FIG. 22 is a view illustrating an example of production of indexinformation with link information;

FIG. 23 is a block diagram showing an example of an internalconfiguration of a production apparatus for image data with taginformation;

FIG. 24 is a flow chart illustrating an outline of the substance ofprocessing of a program corresponding to the production apparatus forimage data with tag information;

FIGS. 25 to 31B are schematic views illustrating different data outputexamples;

FIG. 32 is a block diagram showing an example of an image reproductionsystem;

FIG. 33 is a block diagram showing an example of a configuration wherethe image reproduction system is implemented using a computer;

FIG. 34 is a flow diagram illustrating a corresponding relationshipbetween operations of a viewer and internal processes executed inresponse to the operations;

FIG. 35 is a flow chart illustrating an outline of the subject ofprocessing of a program corresponding to a tag information displaycontrol function;

FIGS. 36A and 36B are schematic views illustrating a correspondingrelationship of a display of a pointer to a main image and acorresponding key image at a certain point of time;

FIG. 37 is a schematic view showing an example of a display of relatedinformation; and

FIGS. 38 to 41B are schematic views showing different display examplesby a guiding function.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following, a preferred embodiment of the present invention isdescribed in connection with production, distribution and reproductionstages of tag information.

It is to be noted that, to particulars which are not shown in theaccompanying drawings or described in the present specification, knowntechniques in the pertaining technical field are applied.

Further, the embodiment described below is a mere example of the presentinvention, and the present invention is not limited to the specificembodiment.

A. Tag Information Production Apparatus

In the following, a tag information production apparatus which canproduce tag information through a realistic amount of work.

A-1. Linear Editing System

The tag information production apparatus can be implemented as a linearediting system.

a. System Configuration

FIG. 1 shows an example of a configuration of the linear editing system.

Referring to FIG. 1, the linear editing system shown includes a mainimage reproduction apparatus 1, an integrating outputting apparatus 3, akey information inputting apparatus 5, a display apparatus 7, a keypattern production apparatus 9, and a key image recording apparatus 11.

The main image reproduction apparatus 1 reproduces a main image recordedon a magnetic tape. The main image reproduction apparatus 1 may beformed, for example, from a VTR (Video Tape Recorder). A main imagereproduced by the main image reproduction apparatus 1 is outputted tothe integrating outputting apparatus 3. A time code reproducedsimultaneously is outputted to the key information inputting apparatus5. It is to be noted that the time code is one of kinds of timeinformation which provides a temporal position of the main image.

The integrating outputting apparatus 3 integrates a key pattern whichprovides embedding information of tag information with the main imageinputted thereto. The integrating outputting apparatus 3 is formed from,for example, a character generator. The key pattern is produced based onkey region information supplied from the key information inputtingapparatus 5 by the integrating outputting apparatus 3. The key regioninformation is given as information relating to the position of the keypattern within a screen and a selection range. The integrated image isoutputted to the display apparatus 7.

FIGS. 2A to 2C show examples of a display screen of the displayapparatus 7. Particularly, FIG. 2A shows an example of a screen (exampleof a working screen) selected as a tag information embedding screen. Theworking screen is given as a main image which is outputted when areproduction action of the main image reproduction apparatus 1 istemporarily stopped.

FIG. 2B shows an example of a working screen where key patterns 21 and23 produced by the integrating outputting apparatus 3 are integratedwith the main image. In the case of the present example, the key pattern21 has such a position and a range that it surrounds an entire vehicledisplayed on the right side on the screen. Meanwhile, the key patterns23 have such positions and ranges that they individually surround twoshoes displayed on the left lower side on the screen.

The shape of a key pattern can be determined by any method such as amethod wherein it is automatically selected by the integratingoutputting apparatus 3, another method wherein the shape is selectedthrough the key information inputting apparatus 5 or a further methodwherein the shape is designated arbitrarily using a draw tool such as,for example, a lasso. In the case of FIGS. 2A to 2C, a square shape anda circular shape are selected through the key information inputtingapparatus 5.

The position and the range at and in which each of the key patterns 21and 23 is to be disposed are determined by the operator throughconfirmation of the integration screen of the display apparatus 7.

In the following, a position and a range on a screen at and in which anindividual key pattern is disposed is referred to as “key region”.

The key information inputting apparatus 5 is used to input keyinformation for defining tag information.

In the present specification, the “tag information” signifiesinformation embedded in a partial region designated arbitrarily.

In the present linear editing system, a “key image” and “indexinformation” are produced as the tag information.

Here, the “key image” signifies an image paired with a main image andindicative of an embedded position of the tag information. Meanwhile,the “index information” is information which represents a correspondingrelationship between a key region and identification information of thekey region with regard to each point of time of the key image.

Meanwhile, the “key information which defines tag information” signifiesinformation necessary for production of key region and indexinformation, for example, identification information of the position,range, time and region and so forth. In the case of the present example,however, the time information is given as a time code from the mainimage reproduction apparatus 1.

The key information inputting apparatus 5 may be formed, for example,from a video editing machine or a computer. In FIG. 1, it is shown thatthe key information inputting apparatus 5 is formed from a computer.

The position and the range of a key pattern are inputted through aninputting apparatus 5A which may be a mouse and so forth. Meanwhile, theinformation for identification of the key pattern is inputted using akey operation of a keyboard, a click operation of the mouse or the like.

It is to be noted that, for the position and the range of a key pattern,a rough position and a rough range of a partial region to be tagged maybe given. Accordingly, no strict positioning is required. Therefore,setting of a key region can be implemented within a range of a realisticamount of work. For example, such setting can be implemented by anamount of work similar to that required for insertion of a caption whichis a general editing operation.

In the case of the present example, a gray scale value (lightness value)is used as a kind of information for identification of a key pattern.Where a gray scale value is given, even if the luminance of a setportion of a key region varies, the set portion can be confirmed.However, there is the possibility that tag information may undergo acompression process upon distribution thereof. Accordingly, preferablythe gray scale value is a value spaced by such a degree that it can bediscriminated even after such compression process. For example, a value“20” is inputted for the key pattern 21 corresponding to a vehicle, andanother value “40” is inputted for the key patterns 23 which correspondto shoes.

The key information inputting apparatus 5 outputs key region informationand region identification information for production of a key image tothe key pattern production apparatus 9, and outputs a time code to thekey image recording apparatus 11. Further, the key information inputtingapparatus 5 outputs index information which is a combination of the keyregion and the region identification information with time informationof a working screen.

The key pattern production apparatus 9 produces a key pattern based onthe key region information and the region identification information setfor the working screen. The key pattern production apparatus 9 may beformed, for example, from a key generator or a character generator. FIG.2C shows an example of a screen which includes the key patterns 21 and23.

It is to be noted that the key pattern production apparatus 9successively produces the same key pattern until after new key regioninformation and new region identification information are received. Thissignifies that key information need not be inputted for all frames. Fromthis, the work amount decreases significantly.

However, if the generation position of the key pattern is fixed, thensome offset appears between the tag information embedded position andthe target object (subject image). Accordingly, the operator wouldconfirm the screen of the display apparatus 7 and suitably modify orre-set the position and the range of the key region.

The key pattern produced by the key pattern production apparatus 9 issupplied together with the time code to the key image recordingapparatus 11, by which it is recorded as a key image having a timelength equal to that of the main image.

It is to be noted that, since the apparatus is formed as a linearediting system, a magnetic tape is used as a recording medium of the keyimage recording apparatus 11.

b. Functional Configuration as a Tag Information Production Apparatus

A functional configuration of the linear editing system as a taginformation production apparatus is described. It is to be noted thatindex information described below includes time information, regionidentification information and object identification information.

FIG. 3 shows a functional configuration as a tag information productionapparatus. Referring to FIG. 3, the tag information production apparatusincludes a time information acquisition section 31, a key region settingsection 33, an identification information application section 35, anobject identification information application section 36, a key imageproduction section 37 and an index information production section 39.

Among the components, the time information acquisition section 31, keyregion setting section 33, identification information applicationsection 35, object identification information application section 36 andindex information production section 39 are implemented through the keyinformation inputting apparatus 5. Meanwhile, the key image productionsection 37 is implemented through the key pattern production apparatus 9and the key image recording apparatus 11.

The time information acquisition section 31 implements a processingfunction of acquiring time information of a main image selected for aworking screen.

The key region setting section 33 implements a processing function ofsetting a key region to a position corresponding to a partial region ina main image specified arbitrarily by the operator.

The identification information application section 35 implements afunction of applying, to the set key region, region identificationinformation for allowing discrimination between the key region and aperipheral region.

The object identification information application section 36 implementsa function of providing identification information for allowingidentification of an object to be specified by the set key region. Inthe case of the present example, while the region identificationinformation is given as a numerical value, the object identificationinformation is given as a character. Therefore, the confirmationoperation can be performed more readily with the object identificationinformation. It is to be noted that also the object identificationinformation may be given as a numerical value.

The key image production section 37 is a functional section ofimplementing a processing function of producing a key image to whichtime information same as that of the main image is added, andcorresponds to a function of producing a key image wherein regionidentification information is applied to a key region set through theworking screen. FIG. 4 illustrates a corresponding relationship betweena key image and key information which defines tag information. From FIG.4, it can be recognized that the key image is produced as an image whosetemporal position corresponds in a one-by-one corresponding relationshipto a main image.

The index information production section 39 corresponds to a processingfunction of producing index information which lists correspondingrelationships among region identification information, time informationand object identification information applied to key regions.

FIG. 5 illustrates a corresponding relationship between indexinformation and key information which defines tag information. As can beseen from FIG. 5, the index information is given as information of thetext type or the XML document type wherein time information, regionidentification information and object identification information arerecorded.

FIG. 6 illustrates an example of index information. The time code isgiven as “hour: minute: second: frame”. Accordingly, FIG. 6 signifiesthat two pieces of tag information are set later than the “00 hour, 00minute, 10 second, 00 frame” of a main image.

In other words, FIG. 6 signifies that two pieces of tag information areset using “20” and “40” of gray scale values.

In FIG. 6, also object identification information is recorded so thatthe operator can confirm with which object (subject image) of a mainimage each piece of region identification information is coordinated.Where object identification information is inserted, it can be utilizedalso for confirmation of whether or not region identification and anintended object coincide with each other.

c. Relationship between a Tagging Operation and Internal Processing

FIG. 7 illustrates a corresponding relationship between a taggingoperation and internal processes executed in response to the taggingoperation. It is to be noted that a tagging operation is inputtedthrough the inputting apparatus 5A of the key information inputtingapparatus 5.

Upon acceptance of a selection operation of the working screen, the keyinformation inputting apparatus 5 issues an instruction to reproduce amain image to the main image reproduction apparatus 1. Further, uponacceptance of a determination operation of the working screen, the keyinformation inputting apparatus 5 issues an instruction to temporarilystop a reproduction operation to the main image reproduction apparatus1.

Further, when an instruction input of a partial region is accepted afterthe determination of the working screen, the key information inputtingapparatus 5 provides key region information to the integratingoutputting apparatus 3. At this time, an integrated image of a keypattern with the working screen (still screen of a main image) isdisplayed on the screen of the display apparatus 7.

Further, when a designation input of region identification informationis accepted with regard to a designated partial region, the keyinformation inputting apparatus 5 provides region identificationinformation to the integrating outputting apparatus 3 and outputs theregion identification information to the key pattern productionapparatus 9. Consequently, a key image is produced. Thereupon, the keyinformation inputting apparatus 5 produces index information based onthe region identification information, time information and objectidentification.

FIG. 8 illustrates the substance of processes of a program to beinstalled into the key information inputting apparatus 5.

Referring to FIG. 8, the program decides whether or not a working screenis determined (S1). If a working screen is determined, then the programacquires a time code as time information (S2). Then, the program acceptssetting of a key region (S3). After a key region is set, the programaccepts setting of region identification information to be coordinatedwith the key region (S4).

Thereafter, the program decides whether or not tagging is ended (S5) andrepeats the setting process of a new key region until after such an endis decided.

Then, after the tagging is ended, the program issues an instruction tooutput the finally determined index information and key image (S6).

The tag information production process is implemented by the keyinformation inputting apparatus 5 executing such a processing program asdescribed above.

d. Effects

By connecting the main image reproduction apparatus 1, integratingoutputting apparatus 3, key pattern production apparatus 9 and key imagerecording apparatus 11 to the key information inputting apparatus 5having the processing functions described above to construct a linearediting process, the system which can record tag information through arealistic amount of work can be implemented.

In other words, a key region can be set not by inputting of text-baseddata such as meta data but by simple visual designation of a region.Further, depending upon the substance of a main image, most of the worksdescribed above can be automated.

Further, a key region set once continues also with regard to succeedingframes until after a new key region is set with regard to a newselection screen (including also a case wherein the same object isused). Therefore, reduction of the amount of inputting work can beachieved.

Further, since a gray scale value is adopted as region identificationinformation, different key regions can be confirmed simply also visuallyfrom a difference in lightness.

Further, since index information which lists corresponding relationshipsbetween time information and information of individual key regions isproduced automatically, a coordinating operation of information withindividual key regions can be carried out efficiently when compared withthat by manual inputting.

A-2. Nonlinear Editing System

The tag information production apparatus can be implemented also as anonlinear editing system.

a. System Configuration

FIG. 9 shows an example of a configuration of the nonlinear editingsystem. The nonlinear editing system is basically formed from a computerby itself.

Referring to FIG. 9, the nonlinear editing system shown includes a CPU(Central Processing Unit) (processor) 41 an inputting apparatus 43, adisplay apparatus 45, a hard disk drive apparatus 47 and a main memory49.

In particular, the main image reproduction apparatus 1 and the key imagerecording apparatus 11 of the linear editing system in FIG. 1 aresubstituted by the hard disk drive apparatus 47. Meanwhile, theintegrating outputting apparatus 3 and the key pattern productionapparatus 9 are implemented through a program executed by the CPU 41using the main memory 49 as a working area.

Naturally, also the key information inputting apparatus 5 is implementedthrough a program which is executed by the CPU 41 using the main memory49 as a working area.

For example, upon determination of a working screen, a main image readout from the hard disk drive apparatus 47 is outputted to the displayapparatus 45 through the main memory 49.

Further, upon setting of a key region, a key pattern produced by anoperation from the inputting apparatus 43 and a main image areintegrated in the main memory 49 and outputted to the display apparatus45. Thereupon, index information inputted through the inputtingapparatus 43 is reflected on the key pattern.

It is to be noted that the key pattern (FIGS. 2C and 4) and the indexinformation (FIG. 5) produced in the main memory 49 are recorded by thehard disk drive apparatus 47.

b. Effects

Also in the case of the nonlinear editing system, similar effects tothose of the linear editing system can be implemented. In other words, asystem which can record tag information through a realistic amount ofwork can be implemented.

A-3. Key Region Inputting Auxiliary Function

a. Region Extraction in Accordance with a Color Condition

In the foregoing description, a key region is set by manual operation.

Here, a technique of setting a region which satisfies a color conditiondesignated in advance by the operator from among partial regionsdesignated within a working screen as a key region is described.

FIGS. 10A and 10B show processing images. In particular, FIGS. 10A and10B illustrate an example wherein shoes of a subject image are set askey regions. It is to be noted that the shoes of the subject image havea color of red.

In this instance, a wide region 51 including the shoes is designated asseen in FIG. 10A. At this time, if a particular color condition (forexample, a red level within a range from a hue value XX to another huevalue YY) is determined as an extraction condition of a key region, thenonly partial regions 53 corresponding to the shoes can be extractedaccurately. It is to be noted that registration of an extractioncondition may be performed before or after designation of a region.

FIG. 11 shows an example of an internal configuration of the key regionsetting section 33 (FIG. 3) where it implements an automatic extractionfunction of a key region in accordance with a color condition.

Referring to FIG. 11, the key region setting section 33 includes adesignated region retaining section 61, a region extraction section 63and an extraction condition retaining section 65. From among thecomponents mentioned, the designated region retaining section 61 is astorage region for storing a region 51 designated through the inputtingapparatus. In other words, the designated region retaining section 61 isa storage region for retaining a region which provides an extractionprocessing range.

The region extraction section 63 is a signal processing section whichextracts a partial region which satisfies a color condition from withinthe region 51 designated from within the main image.

The extraction condition retaining section 65 is a storage region forretaining an extraction condition. In the present example, theextraction condition retaining section 65 is used to retain a colorcondition designated through the inputting apparatus.

This processing function is effective particularly where the differencein hue between a region to be set as a key region and a peripheralregion is great.

It is to be noted that the extraction condition may include not only acolor condition but also a particular shape or pattern. Also in thosecases, simplification of the number of man-hours for inputting can beimplemented similarly as in the case of the region extraction based on acolor condition.

b. Follow-Up Function of a Key Region Based on a Characteristic Amount

In the foregoing description, a key region is set by manual operation.

Here, a technique of storing a characteristic amount of a set key regionand automatically following up the key region in response to movement ofa subject in a main image is described.

FIG. 12 illustrates a processing image. Also FIG. 12 represents anexample wherein shoes of a subject image are set as key regions.

Where the key region follow-up function is adapted, as far as the mainimage for which the key image is set remains within the screen, also thekey region can be moved so as to follow up the movement of the subjectimage in the main image as seen in FIG. 12.

Where the follow-up function described above is incorporated, the amountof operation of the operator is reduced significantly.

FIG. 13 shows an internal configuration of the key region settingsection 33 (FIG. 3) where it implements the automatic follow-up functionof a key region.

Referring to FIG. 13, the key region setting section 33 includes acharacteristic amount storage section 81 and a region follow-up section83. The characteristic amount storage section 81 is a storage region forstoring characteristic amounts (for example, the hue, lightness, shapeand so forth) of a main image corresponding to a key region set throughthe inputting apparatus.

The region follow-up section 83 is a signal processing section whichextracts a partial region which satisfies the stored characteristicamounts from within the main image and outputs the extracted region askey region information. It is to be noted that a known image processingtechnique is applied to the technique for extracting a partial region.

It is to be noted that, while, in the embodiment, manual inputting of akey region is supposed, alternatively such characteristic amount may beregistered in advance independently of a setting operation of a keyregion such that a partial region which satisfies the characteristicamounts is automatically extracted from a main image and set to a keyregion.

For the characteristic amounts, for example, a particular color range, acombination of particular colors, a particular symbol or shape or aparticular character string may be used.

Use of the function described allows automation of considerable part oftag information application operation.

Besides, it is possible to automatically apply tag information to somelive broadcasting program. For example, if a characteristic of anadvertising billboard in a live relay program of sports is registered inadvance, then the advertising billboard appearing on the screen can beautomatically set as a key region. In other words, it is technicallypossible to automatically embed tag information in the advertisingbillboard.

For example, where the advertising billboard has a red color, if a colorrange of red is designated as a characteristic amount, then a partialregion of red in the screen can be set as a key region. It is to benoted that, depending upon the image pickup angle, there is thepossibility that another partial region which satisfies thecharacteristic amount other than an advertising billboard such as aplayer or a spectator may be extracted. In this instance, the operatormay click the advertising billboard so that any other region is exceptedfrom a setting object as a key region.

Alternatively, for example, where an advertising billboard includesyellow characters on the blue background, if this color pattern isdesignated as a characteristic amount, then a partial region whichincludes a similar color pattern can be set as a key region.

Further, for example, if a shape of an advertising billboard or aspecial symbol representing an advertising billboard is set as acharacteristic mount, then a partial region which includes a similarshape or symbol can be set as a key region.

Further, for example, if a character string representative of the nameof an enterprise or a commodity appearing on an advertising billboard isdesignated as a characteristic point, then a partial region whichincludes a similar character string can be set as a key region. In thisinstance, a character recognition technique may be used in combination.

c. Follow-Up Function of a Key Region by an Interpolation Process

In the foregoing description, the automatic follow-up function of a keyregion based on a characteristic amount is described.

Here, a technique of automatically setting, when a locus of movement ofa subject image set as a key region from a certain working screen (startpoint) to another working screen (end point) is estimated, a key regionposition between the two screens by interpolation arithmetic operationusing position information of the key region set on the two screens isdescribed.

FIGS. 14A and 14B illustrate a processing image. Also FIGS. 14A and 14Billustrate examples wherein a vehicle of a subject image is set as a keyregion.

In particular, FIG. 14A illustrates a state wherein a key region is setto a partial region 91 at which a vehicle is positioned on a workingscreen selected as a start point.

FIG. 14B illustrates another state wherein a key region is set to apartial region 93 at which the vehicle is positioned on a working screenselected as an end point.

FIG. 15 shows a locus where the vehicle of the subject image moveslinearly between the two points of time. In this instance, it isestimated that, at an arbitrary point of time between the two points oftime, the vehicle of an object of tagging is positioned at a partialregion 95 taking an acceleration upon movement into consideration.

Accordingly, if a technique for automatically setting an estimatedpartial region as a key region is adopted, then considerably accurateregion designation can be anticipated through a small amount of work.

Further, the amount of arithmetic operation is reduced when comparedwith that in the follow-up process based on a characteristic amountdescribed hereinabove.

FIG. 16 shows an example of an internal configuration of the key regionsetting section 33 (FIG. 3) where it implements an automatic follow-upfunction of a key region.

Referring to FIG. 16, the key region setting section 33 includes a keyregion retaining section 101 and a region position interpolation section103. The key region retaining section 101 is a storage region having aregion 101A for storing the position of a key region set on a workingscreen set as a start point and another region 101B for storing theposition of a key region set on a working screen set as an end point.

The region position interpolation section 103 is a signal processingsection which interpolates the position of the key region between theposition of the key region at the start point and the position of thekey region at the end point in response to the temporal position betweenthe two points.

Where the acceleration or the locus upon movement can be referred to,the region position interpolation section 103 refers to the informationto determine the position of the key region.

In the case of FIG. 15, however, since the vehicle moves linearly, aposition determined based on a ratio in time between the two points isestimated as the position of the key region.

A-4. Modifications

Here, modifications to the linear editing system and the nonlinearediting system are described.

In the foregoing description, upon setting of a key region, a positionand a range are inputted as key region information.

However, only a position may be inputted while it is assumed that thekey region has a fixed configuration (fixed range).

Meanwhile, the key region may have a circular shape as seen in FIG. 17Aor may have an elliptic shape as seen in FIG. 17B. Further, the keyregion may have a square shape as seen in FIG. 18A or may have such apolygonal shape as shown in FIG. 18B or any other polygonal shape.Further, the key region may have any arbitrary configuration. Suchpatterns for a key region as described above are preferably usedselectively by a key operation.

It is to be noted that preferably a function that the configuration of akey region automatically varies as time passes can be selected. Forexample, where a certain key region at a start point is designated as acircular region and the certain region at an end point is designated asan elliptic shape, a morph function of deforming the configuration ofthe key region at intermediate points little by little from the circularconfiguration to the elliptic configuration is applied. The morphfunction is a known function.

Further, while, in the foregoing description, a time code is used astime information representing the reproduction position of a main imageand a key image, also it is possible to use the frame number.

Further, while, in the foregoing description, the size of a main image111 and the size of a key image 113 are equal as seen in FIG. 19A, thesize of the key image 113 may be set to a fraction of the size of themain image 111 as seen in FIG. 19B. For example, the size of the keyimage 113 may be one half (½) or one fourth (¼) the size of the mainimage 111.

For example, where the size of the main image is 1,920×1,080 and thesize of the key image is 960×540, the size of the key image is one halfthat of the main image in both of the horizontal and verticaldirections. Accordingly, the position (x, y)=(10, 20) on the key imageis given as the position (5, 10) on the key image.

The positional relationship between the key image and the main image isconfirmed already upon setting of the key region. Therefore, there is noproblem even if the size of the key image is smaller.

Rather, reduction in size of the key image can reduce the data size.

Further, while, in the foregoing description, a gray scale value(lightness value) is used as region identification information to becoordinated with a key region, a hue value may be used instead.

For example, even where a hue value corresponding to red, blue, yellowor green is used, it can be utilized for identification of a regionwithout any trouble. In addition, also where a plurality of key regionsare visually confirmed on a screen, they can be confirmed readilydepending upon the difference in color.

Further, while, in the foregoing description, a “shoe”, a “vehicle”, an“advertising billboard” or the like is used as an example of a subjectimage which is an embedding object of tag information (setting object ofa key region), a person such as a performer or a player, an animal or aliving thing or the like appearing in a main image may be used as anembedding object of tag information (setting object of a key region).

B. Production Apparatus for Index Information with Link Information andProduction Apparatus of Image Data with Tag Information

B-1. Production Apparatus for Index Information with Link Information

Here, an apparatus for applying link information to index informationproduced formerly is described. In other words, a production apparatusfor index information with link information is described.

If index information produced as described above is used, then a keyregion in the same screen set with regard to a main image can beidentified. However, at this stage, information of an object of aservice cannot be provided to the viewer.

Thus, the inventors of the present invention applies link information tothe index information produced formerly.

FIGS. 20A and 20B show embedded images of ink information.

In particular, FIG. 20A illustrates an embedded image of linkinformation where a URL (Uniform Resource Locator) is embedded by a linkinformation embedding section 121.

FIG. 20B illustrating another embedded image of link information where acommercial message (CM) identifier (local address) is embedded by thelink information embedding section 121. It is to be noted that also itis technically possible to embed a CM itself as link information.

Where link information is embedded in index information in this manner,coordination between a partial region in which the tag information isembedded and the URL or the CM concludes.

FIG. 21 shows an example of an internal configuration of the linkinformation embedding section 121. Referring to FIG. 21, the linkinformation embedding section 121 includes a database 121A and aninformation embedding section 121B.

The database 121A is a region which stores corresponding relationshipsbetween region identification information applied to key region whichprovides partial regions in a main image specified arbitrarily by theoperator and link information for accessing to related information ofthe partial regions.

In other words, the database 121A is a region which stores correspondingrelationships between object identification information and linkinformation or corresponding relationships between region identificationinformation and link information. It is to be noted that thecorresponding relationships are registered in advance. Alternatively, acorresponding relationship may be inputted by manual operation everytime.

The information embedding section 121B is a signal processing sectionwhich refers to the database to embed, in index information which listscorresponding relationships between region identification informationand time information applied to key regions, corresponding linkinformation.

FIG. 22 illustrates an example of production of index information withlink information. More particularly, FIG. 22 illustrates an examplewherein a URL is embedded as link information.

In the case of FIG. 22, two pieces of tag information are set for “00hour: 14 minute: 10 second; 00 frame”. One of the two pieces of taginformation is region identification information “20” which correspondsto one shoe, and the other is region identification information “40”which corresponds to one vehicle.

In the case of the present example, with the region identificationinformation “20”, “http://www.xxx.com/yy/yy” is coordinated as linkformation to the partial region.

Similarly, with the region identification information “40”,“http://www.zzz.com/yy/yy” is coordinated as link formation to thepartial region.

By the use of the link information embedding section 121, a coordinationwork between key regions and link information can be performedefficiently as far as the matching property of correspondingrelationships between information for identifying key regions (forexample, region identification information or object identificationinformation) and link information are assured.

Further, where a main image and related information to the main imageare handled separately from each other in this manner, since maintenanceof them can be carried out separately from each other, variousadvantages can be anticipated.

For example, the maintenance upon change of an address of a relatedinformation site such as upon cancellation of a link of a URL isrequired only for the index information with link information.Therefore, the burden required for maintenance or management can bereduced.

Further, also upon termination of a contract with any advertisingcustomer upon re-distribution (re-broadcasting), it is only necessary toindividually delete the pertaining portion from the index informationwith link information. Therefore, the burden of work decreases.

Further, for example, also when a key region is to be registeredadditionally, although a tagging operation is required, what is to beupdated is only a key image and index information (index informationwith link information). Therefore, there is no necessity to touch withthe main image.

B-2. Production Apparatus for Image Data with Tag Information

If all of index information with link information described above, amain image and a key image become available, then provision of a serviceintended by the inventors of the present invention is permitted. Inother words, it is possible to use a click operation of the viewer whoviews the main image as a trigger to display related information.

On the other hand, if the images and the information can each be handledby itself, then there is the possibility that only the main image may berecorded or duplicated. Where the link information is a CM, recording orduplication only of the main image signifies loss of an opportunity ofadvertisement. This, however, obstructs the development of the mediaindustry in which the CM is a major source of income.

Therefore, the inventors of the present invention propose a technique ofoutputting index information with link information, a main image and akey image in a mutually associated relationship. The technique can thusraise the inevitability that also tag information (index informationwith link information and a key image) is duplicated or recordedintegrally with a main image.

Particularly, if it is required as a condition in duplication orrecording of a main image that all of index information with linkinformation, a main image and a key image be available, then also theadvertisement opportunity can be assured while free enjoyment ispermitted.

Further, even if it is required as a condition to allow a main image tobe produced normally that all of index information with linkinformation, a main image and a key image be available, also theadvertisement opportunity can be assured while free enjoyment ispermitted.

Further, this technique is effective also as a countermeasure of a CMskip where a main image is stored into a hard disk drive apparatus orsome other randomly accessible recording medium and enjoyed later. Thisis because, different from a CM inserted in a gap between main images,the tag information does not make an object of a CM skipping function.

Further, in display of a CM, a click operation of the viewer isrequired. Therefore, a psychological resistance as in a case whereinsuch display is popped up freely irrespective of the will of the vieweris not provided to the viewer. Actually, since the viewer clicks aregion in which tag information is embedded in a main image as a resultthat the viewer is interested in the main image, a higher advertingeffect than that where the CM is included in a continuously given streamof display can be anticipated.

a. System Configuration

In the following, a production apparatus for image data with taginformation is described.

FIG. 23 shows an example of an internal configuration of the productionapparatus for image data with tag information. Referring to FIG. 23, theproduction apparatus for image data with tag information shown includesa link information embedding section 121 (FIG. 21) and a data outputtingsection 123.

The data outputting section 123 is a signal processing section whichoutputs index information with link information, a main image (includinga live image) and a key image in a mutually associated relationship witheach other.

It is to be noted that the production apparatus for image data with taginformation can be implemented also as a program illustrated in FIG. 24.

In particular, the production apparatus for image data with taginformation can be implemented by a process of embedding linkinformation into index information to produce index information withlink information (S11) and another process of outputting the indexinformation with link information, a main image and a key image in amutually associated relationship (S12).

Here, the output destination of the image data with the associated taginformation is a transmission line or a recording medium. Where thetransmission destination is a transmission line, the form ofdistribution may be, for example, downloading, streaming orbroadcasting.

b. Data Format Upon Outputting

In the following, data output examples of image data with taginformation outputted from the data outputting section 123 aredescribed.

b1. Data Output Example 1

FIG. 25 illustrates one of data output examples. Referring to FIG. 25,in the data output example shown, index information with linkinformation is applied to and integrated with a main image and a keyimage both in a non-compressed form so that they are treated as a singlefile. The file produced has a unique extension “cmov” applied thereto. Afile in which a main image, a key image and index information with linkinformation are integrated is hereinafter referred to as “cmov file”.Naturally, it is possible to apply some other extension. For example,“ocv” (Object Click Video) may be applied.

Where a main image, a key image and index information with linkinformation are integrated as in the present data output example, thisis effective to prevent such a situation that only the main image isseparated and utilized by itself.

Where image data are treated in the form of a file, the present dataoutput example is suitable for distribution, for example, in the form ofstreaming, downloading or a recording medium.

b2. Data Output Example 2 FIG. 26 illustrates another one of data outputexamples. Referring to FIG. 26, in the data output example shown, indexinformation with link information is applied to and integrated with amain image and a key image both in a compression-coded form so that theyare treated as a single file. Also in this instance, the uniqueextension “cmov” is applied to the produced file.

The compression code system may be any of a reversible compressioncoding system and an irreversible compression coding system althoughthis applies not only to the present example but also to the otheroutput examples. Generally, for distribution, an irreversiblecompression coding system having a high compression rate is used. On theother hand, for storage of master data or at an editing stage, areversible compression coding system is used.

It is to be noted that, though not shown in FIG. 26, a header is appliedto the produced file. The header is included in the index informationwith link information or is prepared separately.

The header describes the position of the index information with linkinformation placed in the file, the top positions of the main image andthe key image, the number of frames, the picture size (pixel numbers inthe horizontal and vertical directions), the compression system,arrangement information (address offsets) of the index information withlink information/main image/key image in the file and so forth.

Also the present data output example is effective to prevent such asituation that only the main image is separated and utilized because themain image, key image and index information with link information areintegrated with each other. Further, since the data are in a compressioncoded form, the data amount is small. Further, also the present dataoutput example is suitable for distribution in the form of, for example,streaming, downloading and a recording medium where the image data aretreated in the form of a file.

Further, the present data output example allows only the indexinformation with link information to be read out initially or at anarbitrary timing. Accordingly, it is easy to display a table of relatedinformation.

b3. Data Output Example 3

FIG. 27 illustrates a further one of data output examples. Referring toFIG. 27, also in the data output example shown, a main image and a keyimage are compression coded and integrated with each other so that theyare treated as a single file similarly as in the two output examplesdescribed hereinabove. Also in this instance, the unique extension“cmov” is applied.

A unique characteristic of the present output example is that indexinformation with link information is placed in a meta data area of thecompressed key image. The meta data are data which describe informationrelating to the main image. The meta data include, for example, a timecode, copyright information, information of an image pickup person,characters, an image pickup place and so forth.

Also the present data output example is effective to prevent such asituation that only the main image is separated and utilized because themain image, key image and index information with link information areintegrated with each other. Further, since the data are in a compressioncoded form, the data amount is small. Further, also the present dataoutput example is suitable for distribution in the form of, for example,streaming, downloading and a recording medium where the image data aretreated in the form of a file.

It is to be noted that the index information with link information mayotherwise be placed in a meta data area of the compressed main image.

Further, in the case of the present data output example, it is necessaryfor each piece of index information with link information to have onlytag information regarding a corresponding scene (time period).Accordingly, it is possible to make the sizes of individual pieces ofindex information with link information fixed or equal to each other.Therefore, the present data output form is suitable for broadcasting ora content for a long period of time.

b4. Data Output Example 4

FIG. 28 illustrates a still further one of data output examples.Referring to FIG. 25, image data wherein a main image is allocated tothree ones of four data channels and a key image is allocated to theremaining one data channel are compression coded, and index informationwith link information is applied to form a file.

It is to be noted that FIG. 28 illustrates a data output example whereinthree channels for R (red), G (green) and B (blue) are allocated to thethree data channels for a main image. Naturally, the three data channelsmay alternatively be for one luminance data and two color differencedata.

Also in this instance, the integrated file has the unique extension“cmov” applied thereto.

Also in the present data output example, a main image, a key image andindex information with link information are integrated, and this iseffective to prevent such a situation that only the main image isseparated and utilized by itself. Further, since the data are in acompression coded form, the data amount is small. Furthermore, whereimage data are treated in the form of a file, also the present dataoutput example is suitable for distribution, for example, in the form ofstreaming, downloading or a recording medium.

b5. Data Output Example 5

FIG. 29 illustrates a yet further one of data output examples. Referringto FIG. 29, also in the present data output example, image data whereina main image is allocated to three ones of four data channels and a keyimage is allocated to the remaining one data channel are compressioncoded, and index information with link information is applied to form afile. In particular, the unique extension “cmov” is applied to theintegrated file produced.

However, in the present output example, index information with linkinformation is placed in a meta data area of the key image.

Also in the present output example, three channels for R (red), G(green) and B (blue) are allocated to the three data channels for a mainimage. However, one luminance data and two color difference data mayalternatively be applied to the three data channels.

Also the present data output example is effective to prevent such asituation that only the main image is separated and utilized because themain image, key image and index information with link information areintegrated with each other. Further, since the data are in a compressioncoded form, the data amount is small. Further, also the present dataoutput example is suitable for distribution in the form of, for example,streaming, downloading and a recording medium where the image data aretreated in the form of a file.

It is to be noted, in the compression coding process for the presentoutput example, a file wherein a key image (one channel) to which indexinformation with link information is applied and a main image (threechannels) are integrated may be compression coded. In this instance, areversible coding system is used for the compression coding. This iseffective at an editing stage or for storage as master data.

b6. Data Output Example 6

FIGS. 30A and 30B illustrate yet further ones of data output examples.Referring to FIGS. 30A and 30B, in the data output examples shown, amain image is outputted to a main channel while a key image is outputtedto a sub channel. Thereupon, index information with link information isplaced in a meta data area of the main image or the key image.

FIG. 30A shows an example wherein index information with linkinformation is placed in the meta data area of the key image (subchannel) while FIG. 30B shows another example wherein index informationwith link information is placed in the meta data area of the main image(main channel).

The present data output examples are suitable where image data aredistributed in a broadcasting form. Where the present data forms areadopted, also in a broadcasting form, a system wherein relatedinformation of a partial region specified by the viewer can be displayedon a screen can be implemented. It is to be noted that, upon recordingof a main image, if the condition that a sub channel to be paired withthe main channel (main image) is recorded is satisfied, then the presentdata output examples are effective to prevent such a situation that onlythe main image is separated and utilized.

For example, if a main image is broadcast in either of the present dataoutput forms, such application is possible that, although the main imagecan be received in a normal manner by an existing tuner, inhibition ofcopying is effective, or a scramble function becomes effective toinhibit the reception of the main image.

On the other hand, such application can be anticipated with a tunerwhich is compatible with the present data output forms that, when a keyimage and index information with link information are received at thesame time (where a click function is effective), such limitation asinhibition of copying or a scramble function can be canceled.

Further, upon storage into a hard disk drive apparatus or the like, suchapplication is possible that, where a file is stored in the cmov fileformat, inhibition of copying, a scramble function or the like for themain image can be canceled.

b7. Data Output Example 7

FIG. 31A illustrates a yet further one of data output examples.Referring to FIG. 31A, also the present data output example is suitablefor a broadcasting form. However, the present data output example doesnot use a sub channel. In other words, key image and index informationwith link information are placed in a meta data area of a main image inthe main channel.

Where the present data form is adopted, even in the broadcasting form, asystem wherein related information of a partial region specified by theviewer can be displayed on the screen can be implemented. Further, amain image, a key image and index information with link information areintegrated with each other, and this is effective to prevent such asituation that only the main image is separated and utilized by itself.

Also it is effective to record related information (for example, a CM)representative of a linking destination of link information in the metadata area together with index information with link information as seenin FIG. 31B.

It is to be noted that the link information is given as informationrepresentative of on what numbered page of related information (forexample, a CM) the linking destination is or identification information(for example, a number) unique to individual related information.

In this instance, since related information of a linking designation isintegrated, there is no necessity to acquire the related informationthrough the Internet. Accordingly, the present data output form iseffective where the reproduction apparatus is not connected to theInternet. Also where the reproduction apparatus is connected to theInternet, the present data output form is effective where thecommunication band is not sufficient or the communication condition isnot preferable.

It is to be noted that the method of integrating related informationrepresentative of a linking designation of link information with a mainimage in this manner is effective also to the data output examples of afile of the cmov file format described hereinabove.

C. Tag Information Display Control Apparatus and Information ProcessingApparatus (Display Apparatus)

Here, processing action when image data with tag information isreproduced, that is, processing action executed by an informationprocessing apparatus on the viewer side, is described.

It is to be noted that the information processing apparatus may be a taginformation display control apparatus or an apparatus of any commodityform only if it can implement equivalent functions to those of the taginformation display control apparatus by software.

For example, the information processing apparatus may be a computer, amagnetic disk apparatus, an optical disk player, a set top box, a homeserver or any other large-capacity storage apparatus, a displayapparatus, a game machine, a digital camera, a portable informationterminal, a portable telephone set or the like.

Image data with tag information by the image processing apparatus can bereproduced by two reproduction modes.

One of the reproduction modes is reproduction of the accumulation type.

The reproduction of the accumulation type is reproduction which isstarted in a state wherein entire image data with tag information arestored on a recording medium

For example, image data with tag information (cmov file) distributed ina form wherein they are recorded on an optical disk or some otherlarge-capacity medium may be reproduced.

Alternatively, for example, image data with tag information (cmov file)downloaded through the Internet may be reproduced from a local medium.

Upon such reproduction, a main image and a key image are reproduced insynchronism with each other. However, the key image is not displayed onthe display screen.

Alternatively, for example, a main image, a key image and indexinformation with link information received in a broadcasting form or astreaming form may be reproduced from a recording medium on which imagedata with tag information (cmov file) reconstructed from the main image,key image and index information with link information are recoded. It isto be noted that the recording medium may be a portable recording mediumor a local medium incorporated in the reproduction apparatus.

Meanwhile, the other reproduction mode is reproduction of the streamingtype.

The reproduction of the streaming type is reproduction of a streamingfile in the downloaded order.

It is to be noted that, in the reproduction of the streaming type, indexinformation with link information is downloaded first and stored into amain memory. Thereafter, a main image and a key image are downloaded ina reproduction order and stored into the memory, and then the main imageand the key image are reproduced in synchronism with each other.

Also in this instance, the key image is not reproduced.

C-1. Image Reproduction System

In the following, an image reproduction system which is used commonlyfor both of reproduction of the storage type and reproduction of thestreaming type is described.

a. Functional Configuration

FIG. 32 shows an example of a configuration of the image reproductionsystem.

Referring to FIG. 32, the image reproduction system shown includes a taginformation display control apparatus 131, an inputting apparatus 133, areproduction apparatus 135, and a display apparatus 137.

First, the components of the image reproduction system other than thetag information display control apparatus 131 are described.

The inputting apparatus 133 is used to input an operation instructionfrom a viewer who enjoys image data with tag information.

The reproduction apparatus 135 is used to reproduce image data with taginformation from a recording region. A reproduction mechanism suitablefor the recording medium is used as the reproduction apparatus 135. Itis to be noted that a large-capacity recording medium is used forreproduction of the accumulation type while a main memory (semiconductorstorage device) is used for reproduction of the streaming type.

The display apparatus 137 displays a main image. Various display devicescan be used for the display apparatus 137. For example, a CRT displayunit, a flat display unit, or a projection type display apparatus (forexample, a liquid crystal panel, or a DMD (registered trademark) device)is used for the display apparatus 137.

The tag information display control apparatus 131 is a control unitwhich controls the peripheral apparatus described above to controldisplay of a main image and display of related information in adesignated region. The tag information display control apparatus 131 canbe implemented by any of software and hardware.

Here, a functional configuration of the tag information display controlapparatus 131 is described.

The tag information display control apparatus 131 includes areproduction control section 141, an index information reading outsection 143, a pointer movement acceptance section 145, a pointerposition acquisition section 147, and a pointer position decisionsection 149. The tag information display control apparatus 131 furtherincludes an identification information reading out section 151, a linkinformation reading out section 153 and a related information displaycontrol section 155.

The reproduction control section 141 is a control apparatus whichcauses, when a reproduction instruction of a main image is accepted, akey image having time information (for example, a time code) same asthat of the main image to be reproduced synchronously. In other words,the reproduction control section 141 controls synchronous reproductionof a main image and a key image.

The index information reading out section 143 is a signal processingapparatus which reads out index information with link informationassociated with a main image and a key image.

The pointer movement acceptance section 145 is a signal processingapparatus which accepts movement of a pointer within the screen by aviewer of the main image through the inputting apparatus 133.

The pointer position acquisition section 147 is a signal processingapparatus which acquires the position of the pointer in the screen.

The pointer position decision section 149 is a signal processingapparatus which refers to a key image reproduced synchronously with amain image to decide whether or not the pointer position is within thekey region. In other words, the pointer position decision section 149performs a matching decision between a key position having the sametemporal position as that of the main image being currently displayedand the pointer.

The identification information reading out section 151 is a signalprocessing apparatus which reads out region identification informationset to a key region when the pointer position is within the key region.For example, where the pointer belongs to a key region corresponding toa shoe, the gray scale value of the key region is read out.

The link information reading out section 153 is a signal processingapparatus which refers to index information with link information basedon time information of a key image referred to and region identificationinformation read out to read out link information corresponding to theregion identification information.

The related information display control section 155 is a signalprocessing apparatus which acquires related information from a linkdestination specified with read out link information and displays theacquired related information.

For example, where the link information is given as a URL, the relatedinformation display control section 155 acquires necessary relatedinformation from a web site accessed through a communication functionnot shown and provides the acquired related information to the displayapparatus 137.

On the other hand, for example, where the link information isidentification information of related information received or storedtogether with a main image, the related information display controlsection 155 acquires necessary related information through thereproduction apparatus 135 and provides the acquired related informationto the display apparatus 137.

b. Particular Examples

FIG. 33 shows an example of a configuration where the image reproductionsystem is implemented from a computer.

Referring to FIG. 33, the image reproduction system shown includes a CPU(processor) 161, an inputting apparatus 163, a display apparatus 165, ahard disk drive apparatus 167 and a main memory 169.

In particular, the processing functions of the tag information displaycontrol apparatus 131 shown in FIG. 33 are implemented through a programexecuted by the CPU 161.

For example, a decision process of deciding whether or not thecoordinate position of the pointer designated by the viewer is includedin a key region and a reading out process of reading out linkinformation based on region identification information applied to thekey region at which the pointer is positioned are executed by the CPU161.

It is to be noted that the image reproduction system shown in FIG. 33 isconfigured so as to perform reproduction of the accumulation type. Inother words, the image reproduction system is shown configured such thata main image, a key image and index information with tag information areread out from the hard disk drive apparatus 167 to the main memory 169.

Accordingly, where the image reproduction system is configured otherwiseso as to perform reproduction of the streaming type, processing advancesin a state wherein a main image, a key image and index information withtag information are stored directly in the main memory 169.

C-2. Relationship between an Operation of a Viewer and an InternalProcess

FIG. 34 illustrates a corresponding relationship between an operation ofa viewer and internal processes executed in response to the operation.It is to be noted that an operation of the viewer is inputted throughthe inputting apparatus 163 (133).

When an operation for reproduction of a main image is accepted, the CPU161 issues an instruction to read in index information with linkinformation and reproduce (display) a main image. Thereupon, the CPU 161issues also an instruction to perform synchronous reproduction of a keyimage. Further, the CPU 161 starts up a guiding function hereinafterdescribed. The guiding function is a function of notifying the viewer ofthe presence of an embedded key region and assisting an appropriateoperation for the key region.

On the other hand, when the CPU 161 accepts a clicking operation of atagged object (subject image), it detects the click coordinates(acquires the pointer position) and acquires region identificationinformation corresponding to the click coordinates from a key image towhich the same time is applied. For example, the CPU 161 acquires a grayscale value.

Thereafter, the CPU 161 refers to index information with tag informationcoordinated with the same time or a time interval including the timeusing the region identification information to acquire relatedinformation from corresponding link information. The acquired relatedinformation is displayed on the screen of the display apparatus 165. Thedisplay may be a superimposed display on the main image or may be achild screen display.

FIG. 35 illustrates the substance of processes of a programcorresponding to the tag information display control function describedabove.

Referring to FIG. 35, the program is started with a reading out processof index information with tag information (S21). Then, the programcontrols synchronous reproduction of a main image and a key image (S22).

Thereafter, the program accepts an input of pointer movement displayedon the screen of the display apparatus 165 and decides the pointerposition (S23, S24).

FIG. 36A shows an example of display of a pointer 171. FIG. 36A, thepointer 171 is positioned at a position of a shoe of a subject image.

Then, the program decides whether or not the pointer position is withina key region (S25). In the decision process, the program refers to a keyimage at time same as that of the main image. FIG. 36B illustrates a keyimage paired with the main image. Again, the key image is for internalprocessing and is not actually displayed to the viewer.

FIGS. 36A and 36B show examples of an image set for the “00 hour: 14minute: 30 second: 12 frame”. In the examples, a key region (gray scalevalue=20) of a square shape is set at a position corresponding to avehicle, and another key region (gray scale value=40) of a circularshape is set at a position corresponding to each of the shoes.

The decision process is executed depending upon whether or not thecoordinates (X, Y) of the pointer are positioned in a key region.

If it is decided in the decision process that the pointer position isnot within any key region (if a negative result is obtained), then theprogram returns to a state wherein movement of the pointer is accepted.This is because the pointer designates a region in which no taginformation is embedded.

On the other hand, if it is decided in the decision process that thepointer position is in a key region (if an affirmative result isobtained), then the program accesses index information with linkinformation based on the region identification information of the keyregion to read out link information coordinated with the correspondingregion identification information with regard to the same time as thatupon the operation (S26).

Thereafter, the program acquires related information given by the linkinformation and controls the display apparatus 165 to display therelated information on the screen (S27).

FIG. 37 shows an example of the display of related information. In FIG.37, a URL address (http://www.xxxx.com/yy/yy) representative of a sourceof the related information is displayed at an upper portion of thescreen.

Meanwhile, at a central portion of the screen, the shoe designated withthe pointer is displayed in a large screen size. Simultaneously, a price(in the example shown, “$98”) is displayed at a left upper portion ofthe screen.

Naturally, the display window of the related information can be closedby an operation of the viewer.

Thereafter, the program decides whether or not reproduction of a mainimage comes to an end (S28). If reproduction continues (if a negativeresult is obtained), then the program returns to the input acceptingstate of a pointer position in preparation for designation of a furtherkey region.

On the other hand, if an end of reproduction is confirmed (if anaffirmative result is obtained), then the program ends the displaycontrol function of tag information.

Such a processing program as described above is executed by the CPU 161to execute a tag information display process.

C-3. Guiding Function

As described above, by reproducing image data with tag information,related information embedded at a pointer position can be confirmed onthe screen as a result of a subjective operation of the viewer.

Here, a display function suitable to assist the viewer to designate akey region is described.

a. Guiding Function 1

While a key image is originally prepared only for a decision process, itcan be utilized also to assist the viewer in a screen operation.

For example, a function (key region display control section) ofdisplaying, when a key region is included in a key image reproducedsynchronously with a main image, a partial region of the main imagecorresponding to the key region in such a display mode that the viewercan recognize the partial region can be incorporated as a function ofthe tag information display control apparatus 131.

FIG. 38 shows a display example where this display function is adopted.In FIG. 38, a main image and a key image are displayed in an integratedstate.

In this instance, the partial region of the main image in which taginformation is embedded may be displayed with the luminance thereofchanged. Where a luminance different from the original luminance of themain image is provided to the key image, the partial region in which taginformation is embedded can be discriminated readily.

It is to be noted that the luminance change may be provided in responseto region identification information (gray scale value or hue value)provided to each key region which forms a key image.

Alternatively, the same luminance change may be provided to all of thekey regions without depending upon the difference in regionidentification information provided to the key regions. This is becausewhat is necessary to the viewer is to identify a partial region in whichtag information is embedded but not to identify the difference betweenregions.

Incidentally, the luminance change may be performed not only by raisingthe luminance with respect to the original luminance level but also bylowering the luminance or by periodically turning on and off theluminance. Here, the luminance change may be provided by a level ofseveral percent of the luminance of the main image.

It is to be noted that the method of uniformly changing the luminance ina partial region in which tag information is embedded deteriorates thepicture quality of the main image and makes an obstruction to theenjoyment.

Accordingly, the display mode which relies upon the luminance change ispreferably executed only when the display mode in which the luminance ofa partial region is changed is selected by the viewer. Naturally, theluminance of a region in which tag information is embedded mayalternatively be changed normally.

Meanwhile, as a technique of changing the display mode of a partialregion in which tag information is embedded to notify the viewer, also amethod may be available wherein the hue of a partial region to be usedfor display is changed.

For example, the change of the hue may be provided in response to regionidentification information (hue value or gray scale value) provided toeach key region which forms the key image.

Alternatively, the same hue change may be provided to all key regionswithout depending upon the difference of the region identificationinformation provided to the key regions. For example, a red color may beprovided or a framework of red may be displayed for all of entirepartial regions in which tag information is embedded. Naturally, anarbitrary color may be used for the display.

Also the display mode which relies upon the change in hue is preferablyexecuted only when the display mode in which the luminance of a partialregion is changed is selected by the viewer. Naturally, the luminance ofa region in which tag information is embedded may alternatively bechanged normally.

b. Guiding Function 2

The guiding function 1 described above changes the display mode of apartial region in which tag information is embedded without dependingupon the position of the pointer. However, they may possibly disturbenjoyment of a main image.

Therefore, a function (tag region position notification section) ofdisplaying, where it is decided that the pointer position is within arange of a key region or approaches a key region, a partial region of amain image in which tag information is embedded in such a display modethat the viewer can recognize the partial region may be incorporated asa function of the tag information display control apparatus 131.

FIGS. 39A and 39B show display examples where this display function isadopted. Also in the case of FIGS. 39A and 39B, tag information isembedded in shoes of a subject image similarly as in the other examplesdescribed hereinabove.

In particular, FIG. 39A shows a display example wherein a pointer 173 isnot positioned in any key region set on a key screen. At this time, nochange is found on the display screen.

Accordingly, the viewer can determine that no tag information isembedded at the current pointer position. Further, no obstacle is madeto enjoyment of the main image.

FIG. 39B shows a display example wherein the pointer 173 is positionedin a key region set on a key image. In this instance, the display modeof the key region corresponding to the shoe changes. For example, theluminance changes or the hue changes similarly as in the case of theguiding function 1.

Depending upon the change, the viewer can visually confirm that taginformation is embedded at the current pointer position.

It is to be noted that, in the present display example, the change ofthe display mode of the main image is performed only in the key regionscorresponding to the shoes, but does not occur in the key regioncorresponding to the vehicle.

Therefore, deterioration of the picture quality of the main image whentag information is displayed can be minimized. The present displayexample is effective particularly where tag information is embedded in alarge number of regions.

c. Guiding Function 3

In the two guiding functions described above, the display mode of a mainimage is changed. However, this may possibly deteriorate the enjoymentof the main image by the viewer.

Therefore, a technique of incorporating a function (pointer shapechanging section) of changing the display mode of the pointer to bepositioned in a partial region of a main image in which tag informationis embedded when it is decided that the pointer position is within arange of a key region as a function of the tag information displaycontrol apparatus 131 is described.

FIGS. 40A, 40B and 41A, 41B show different display examples where thedisplay function just described is adopted.

In particular, FIGS. 40A and 41A shows display examples where a pointer175 is not positioned in any key region set on a key screen. In thisinstance, the pointer 175 is displayed in a basic size and a basicshape.

FIGS. 40B and 41B show display examples where the pointer 175 ispositioned in a key region set on a key screen.

In FIG. 40B, a pointer 177 is displayed in an equal size but in adifferent display mode such as a luminance or a hue.

In FIG. 41B, a pointer 179 is displayed in the same luminance and thesame hue but in a changed display mode.

In this display function, since only the display mode of the pointer ischanged, there is no influence on the picture quality of the main image.

Further, since only the display mode of the pointer is changed, also theamount of arithmetic operation is small. Furthermore, since the displaymode of the pointer which attracts attention of the viewer is changed,the viewer can confirm the key region readily.

D. Other Embodiments

a. Programs corresponding to the functions described hereinabove may bedistributed through a network or stored in or on and distributedtogether with a storage medium. The storage medium for distribution maybe a magnetic storage medium, an optical recording medium, asemiconductor storage medium or any other storage medium.

b. Various modifications can be made to the embodiment describedhereinabove within the spirit and scope of the present invention. Alsovarious alterations and applications can be made by creation orcombination based on the disclosure of the present invention.

1. A production apparatus for index information with link information,comprising: a database in which a corresponding relationship betweenregion identification information applied to a key region which providesa partial region in a main image specified arbitrarily by an operatorand link information for accessing to related information of the partialregion is listed; and an information embedding section configured torefer to said database to embed, into index information in which acorresponding relationship between the region identification informationapplied to the key region and time information is listed, thecorresponding link information.
 2. The production apparatus for indexinformation with link information according to claim 1, wherein saiddatabase records the corresponding relationship in combination with timeinformation.
 3. A production apparatus for index information with linkinformation, comprising: a database in which a correspondingrelationship between object identification information applied to a keyregion which provides a partial region in a main image specifiedarbitrarily by an operator and link information for accessing to relatedinformation of the partial region is listed; and an informationembedding section configured to refer to said database to embed, intoindex information in which a corresponding relationship between theobject identification information applied to the key region and timeinformation is listed, the corresponding link information.
 4. Theproduction apparatus for index information with link informationaccording to claim 3, wherein said database records the correspondingrelationship in combination with time information.
 5. A productionapparatus for image data with tag information, comprising: a database inwhich a corresponding relationship between region identificationinformation applied to a key region which provides a partial region in amain image specified arbitrarily by an operator and link information foraccessing to related information of the partial region is listed; aninformation embedding section configured to refer to said database toembed, into index information in which a corresponding relationshipbetween the region identification information applied to the key regionand time information is listed, the corresponding link information; anda data outputting section configured to output the index informationwith link information in which the link information is embedded, themain image and a key image to which the region identificationinformation which specifies the key region in combination with the timeinformation is applied in a mutually associated relationship.
 6. Theproduction apparatus for image data with tag information according toclaim 5, wherein the outputting destination of said data outputtingsection is a transmission line.
 7. The production apparatus for imagedata with tag information according to claim 5, wherein said dataoutputting section outputs the index information with link information,main image and key image in a broadcasting form.
 8. The productionapparatus for image data with tag information according to claim 5,wherein said data outputting section outputs the index information withlink information, main image and key image in a distribution form. 9.The production apparatus for image data with tag information accordingto claim 5, wherein the outputting destination of said data outputtingsection is a recording medium.
 10. The production apparatus for imagedata with tag information according to claim 5, wherein the main imageis a live image.
 11. The production apparatus for image data with taginformation according to claim 5, wherein said data outputting sectionoutputs the index information with link information, main image and keyimage collectively as a file.
 12. The production apparatus for imagedata with tag information according to claim 5, wherein said dataoutputting section applies the index information with link informationto two compression coded data obtained by compression coding the mainimage and the key image to produce a file.
 13. The production apparatusfor image data with tag information according to claim 5, wherein saiddata outputting section applies the index information with linkinformation to single compression coded data obtained by compressioncoding image data wherein three of four data channels are allocated tothe main image and the remaining one data channel is allocated to thekey image to produce a file.
 14. The production apparatus for image datawith tag information according to claim 5, wherein said data outputtingsection applies the index information with link information tocompression coded data obtained by compression coding image data whereinthree of four data channels are allocated to the main image and theremaining one data channel is allocated to the key image in which theindex information with link information is embedded as incidentalinformation to produce a file.
 15. The production apparatus for imagedata with tag information according to claim 5, wherein said dataoutputting section outputs the main image to a main channel andsimultaneously outputs the key image to a sub channel while the indexinformation with link information image is placed in a meta data area ofthe main image or the key image.
 16. The production apparatus for imagedata with tag information according to claim 5, wherein said dataoutputting section outputs the main image to a main channel while thekey image and the index information with link information are placed ina meta data area of the main image.
 17. The production apparatus forimage data with tag information according to claim 5, wherein said dataoutputting section outputs a body of related information which makes adestination of linking of the link information embedded in the indexinformation with link information in an associated relationship with theindex information with link information, main image and key image.
 18. Aproduction apparatus for image data with tag information, comprising: adatabase in which a corresponding relationship between objectidentification information applied to a key region which provides apartial region in a main image specified arbitrarily by an operator andlink information for accessing to related information of the partialregion is listed; an information embedding section configured to referto said database to embed, into index information in which acorresponding relationship between the object identification informationapplied to the key region and time information is listed, thecorresponding link information; and a data outputting section configuredto output the index information with link information in which the linkinformation is embedded, the main image and a key image to which theregion identification information which specifies the key region incombination with the time information is applied in a mutuallyassociated relationship.
 19. The production apparatus for image datawith tag information according to claim 6, wherein the outputtingdestination of said data outputting section is a transmission line. 20.The production apparatus for image data with tag information accordingto claim 6, wherein said data outputting section outputs the indexinformation with link information, main image and key image in abroadcasting form.
 21. The production apparatus for image data with taginformation according to claim 6, wherein said data outputting sectionoutputs the index information with link information, main image and keyimage in a distribution form.
 22. The production apparatus for imagedata with tag information according to claim 6, wherein the outputtingdestination of said data outputting section is a recording medium. 23.The production apparatus for image data with tag information accordingto claim 6, wherein the main image is a live image.
 24. The productionapparatus for image data with tag information according to claim 6,wherein said data outputting section outputs the index information withlink information, main image and key image collectively as a file. 25.The production apparatus for image data with tag information accordingto claim 6, wherein said data outputting section applies the indexinformation with link information to two compression coded data obtainedby compression coding the main image and the key image to produce afile.
 26. The production apparatus for image data with tag informationaccording to claim 6, wherein said data outputting section applies theindex information with link information to single compression coded dataobtained by compression coding image data wherein three of four datachannels are allocated to the main image and the remaining one datachannel is allocated to the key image to produce a file.
 27. Theproduction apparatus for image data with tag information according toclaim 6, wherein said data outputting section applies the indexinformation with link information to compression coded data obtained bycompression coding image data wherein three of four data channels areallocated to the main image and the remaining one data channel isallocated to the key image in which the index information with linkinformation is embedded as incidental information to produce a file. 28.The production apparatus for image data with tag information accordingto claim 6, wherein said data outputting section outputs the main imageto a main channel and simultaneously outputs the key image to a subchannel while the index information with link information image isplaced in a meta data area of the main image or the key image.
 29. Theproduction apparatus for image data with tag information according toclaim 6, wherein said data outputting section outputs the main image toa main channel while the key image and the index information with linkinformation are placed in a meta data area of the main image.
 30. Theproduction apparatus for image data with tag information according toclaim 6, wherein said data outputting section outputs a body of relatedinformation which makes a destination of linking of the link informationembedded in the index information with link information in an associatedrelationship with the index information with link information, mainimage and key image.
 31. A production method for index information withlink information, comprising the step of: referring to a database inwhich a corresponding relationship between region identificationinformation applied to a key region which provides a partial region in amain image specified arbitrarily by an operator and link information foraccessing to related information of the partial region is listed toembed, into index information in which a corresponding relationshipbetween the region identification information applied to the key regionand time information is listed, the corresponding link information. 32.A production method for index information with link information,comprising the step of: referring to a database in which a correspondingrelationship between object identification information applied to a keyregion which provides a partial region in a main image specifiedarbitrarily by an operator and link information for accessing to relatedinformation of the partial region is listed to embed, into indexinformation in which a corresponding relationship between the objectidentification information applied to the key region and timeinformation is listed, the corresponding link information.
 33. Aproduction method for image data with tag information, comprising thesteps of: referring to a database in which a corresponding relationshipbetween region identification information applied to a key region whichprovides a partial region in a main image specified arbitrarily by anoperator and link information for accessing to related information ofthe partial region is listed to embed, into index information in which acorresponding relationship between the region identification informationapplied to the key region and time information is listed, thecorresponding link information; and outputting the index informationwith link information in which the link information is embedded, themain image and a key image to which the region identificationinformation which specifies the key region in combination with the timeinformation is applied in a mutually associated relationship.
 34. Aproduction method for image data with tag information, comprising thesteps of: referring to a database in which a corresponding relationshipbetween object identification information applied to a key region whichprovides a partial region in a main image specified arbitrarily by anoperator and link information for accessing to related information ofthe partial region is listed to embed, into index information in which acorresponding relationship between the object identification informationapplied to the key region and time information is listed, thecorresponding link information; and outputting the index informationwith link information in which the link information is embedded, themain image and a key image to which the region identificationinformation which specifies the key region in combination with the timeinformation is applied in a mutually associated relationship.
 35. Acomputer-readable recording medium in which a program is recorded, theprogram comprising the step of: referring to a database in which acorresponding relationship between region identification informationapplied to a key region which provides a partial region in a main imagespecified arbitrarily by an operator and link information for accessingto related information of the partial region is listed to embed, intoindex information in which a corresponding relationship between theregion identification information applied to the key region and timeinformation is listed, the corresponding link information.
 36. Acomputer-readable recording medium in which a program is recorded, theprogram comprising the step of: referring to a database in which acorresponding relationship between object identification informationapplied to a key region which provides a partial region in a main imagespecified arbitrarily by an operator and link information for accessingto related information of the partial region is listed to embed, intoindex information in which a corresponding relationship between theobject identification information applied to the key region and timeinformation is listed, the corresponding link information.
 37. Acomputer-readable recording medium in which a program is recorded, theprogram comprising the steps of: referring to a database in which acorresponding relationship between region identification informationapplied to a key region which provides a partial region in a main imagespecified arbitrarily by an operator and link information for accessingto related information of the partial region is listed to embed, intoindex information in which a corresponding relationship between theregion identification information applied to the key region and timeinformation is listed, the corresponding link information; andoutputting the index information with link information in which the linkinformation is embedded, the main image and a key image to which theregion identification information which specifies the key region incombination with the time information is applied in a mutuallyassociated relationship.
 38. A computer-readable recording medium inwhich a program is recorded, the program comprising the steps of:referring to a database in which a corresponding relationship betweenobject identification information applied to a key region which providesa partial region in a main image specified arbitrarily by an operatorand link information for accessing to related information of the partialregion is listed to embed, into index information in which acorresponding relationship between the object identification informationapplied to the key region and time information is listed, thecorresponding link information; and outputting the index informationwith link information in which the link information is embedded, themain image and a key image to which the region identificationinformation which specifies the key region in combination with the timeinformation is applied in a mutually associated relationship.